Implementing Pagination and Large Dataset Handling

Course: Android App Development

Topic: Implementing Pagination and Large Dataset Handling

Course Link: Android App Development Course

Demo Link: Register for a Course Demo

Website: JustAcademy

1. Introduction to Pagination

Pagination is a technique for retrieving, processing, and displaying a large collection of records in smaller portions called pages. Instead of loading thousands of records at once, an Android application loads an initial group and retrieves additional records when needed.

For example, a shopping application may contain 20,000 products. Loading every product immediately can consume memory, increase network traffic, and delay the first screen. With pagination, the app might load 20 products initially and fetch additional products as the user scrolls.

Pagination is useful for product catalogs, social media feeds, chat histories, transaction records, news feeds, student directories, and search results.

2. Learning Objectives

• Understand pagination and large dataset challenges in Android applications.
• Learn page-number, offset-based, and cursor-based pagination.
• Implement basic manual pagination using Kotlin and RecyclerView.
• Understand Android Jetpack Paging 3 architecture.
• Create a PagingSource, Pager, PagingConfig, and PagingData stream.
• Display paginated data using PagingDataAdapter.
• Handle loading states, errors, retry actions, and refresh operations.
• Understand Room database integration and RemoteMediator.
• Apply performance and memory-management best practices.

3. Why Large Datasets Need Special Handling

Loading a large dataset all at once creates several potential problems.

4. How Pagination Works

1. Initial request: The app requests the first page of records.
2. Display data: The returned records are displayed in RecyclerView.
3. Detect demand: As the user approaches the end of the loaded list, the app requests more records.
4. Append results: The new records are added to the displayed collection.
5. Continue or finish: Additional pages are loaded until the server indicates that no more data is available.

The exact implementation depends on how the backend API or database identifies the next portion of data.

5. Types of Pagination

5.1 Page-Number-Based Pagination

The client requests a page number and a page size. The server returns the corresponding records.

GET /api/products?page=1&size=20
GET /api/products?page=2&size=20

This approach is easy to understand and is commonly used by APIs that divide records into numbered pages.

5.2 Offset-Based Pagination

The client specifies how many records to skip and how many to retrieve.

GET /api/products?offset=0&limit=20
GET /api/products?offset=20&limit=20

The first request starts at record zero and retrieves 20 records. The next request skips those records and retrieves the next group. Large offsets can become inefficient for some databases, and records inserted or deleted between requests can shift the results.

5.3 Cursor-Based Pagination

The server returns a cursor or continuation token that identifies where the next request should continue. The client sends that token to retrieve the next group.

GET /api/products?limit=20
GET /api/products?limit=20&cursor=eyJpZCI6MjB9

The cursor shown is illustrative. A real cursor must come from the API response and follow the backend's contract.

Cursor-based pagination is often useful for frequently changing feeds because it can provide more stable continuation than a large numeric offset, provided the backend implements it correctly.

Comparison of Pagination Strategies

6. Manual Pagination with RecyclerView

For a small application, manual pagination can be implemented using a RecyclerView scroll listener. The app tracks the current page, detects when the user approaches the end, and requests the next page.

However, manual implementations must manage loading state, errors, duplicate requests, refreshes, cancellation, and changes to the data source. For production applications with substantial paging requirements, Android Jetpack Paging 3 is often a better choice.

7. Example Data Model

Consider an application that displays products retrieved from a paginated API.

data class Product(
    val id: Int,
    val name: String,
    val price: Double
)

data class ProductPage(
    val items: List<Product>,
    val hasMore: Boolean
)

The ProductPage class is an example response model. Your actual model should match the fields returned by your backend, including its page number, total count, next cursor, or continuation URL when applicable.

8. Manual Pagination: Repository and API Contract

Assume the repository exposes a suspend function that requests a page from the server. The network implementation is specific to the API, so the following code defines the expected contract rather than a complete Retrofit service.

interface ProductRepository {
    suspend fun getProducts(
        page: Int,
        pageSize: Int
    ): ProductPage
}

A Retrofit implementation would define the appropriate endpoint and deserialize the server response into ProductPage. The endpoint's parameter names and response fields must match the real API.

9. Manual Pagination: ViewModel Example

The following example demonstrates the core logic for page-number pagination. It prevents simultaneous requests, stores the loaded records, exposes loading and error states, and allows the user to retry after a failure.

class ProductViewModel(
    private val repository: ProductRepository
) : ViewModel() {

    private val _products =
        MutableLiveData<List<Product>>(emptyList())
    val products: LiveData<List<Product>> = _products

    private val _isLoading = MutableLiveData(false)
    val isLoading: LiveData<Boolean> = _isLoading

    private val _error = MutableLiveData<String?>(null)
    val error: LiveData<String?> = _error

    private var nextPage = 1
    private val pageSize = 20
    private var hasMore = true

    fun loadNextPage() {
        if (_isLoading.value == true || !hasMore) return

        viewModelScope.launch {
            _isLoading.value = true
            _error.value = null

            try {
                val result = repository.getProducts(
                    page = nextPage,
                    pageSize = pageSize
                )

                _products.value = _products.value.orEmpty() +
                    result.items

                hasMore = result.hasMore
                if (result.items.isNotEmpty()) {
                    nextPage++
                } else {
                    hasMore = false
                }
            } catch (exception: Exception) {
                _error.value =
                    exception.message ?: "Unable to load products"
            } finally {
                _isLoading.value = false
            }
        }
    }
}

This is a teaching example. A production implementation should also consider cancellation, lifecycle-safe error handling, duplicate records, authentication errors, and resetting pagination after a refresh. The repository must correctly report whether more records are available.

10. Detecting the End of a RecyclerView List

A scroll listener can request more records when the user approaches the last few visible items. This is a simplified example for a vertical LinearLayoutManager.

recyclerView.addOnScrollListener(
    object : RecyclerView.OnScrollListener() {

        override fun onScrolled(
            recyclerView: RecyclerView,
            dx: Int,
            dy: Int
        ) {
            super.onScrolled(recyclerView, dx, dy)

            if (dy <= 0) return

            val layoutManager =
                recyclerView.layoutManager as? LinearLayoutManager
                    ?: return

            val totalItems = layoutManager.itemCount
            val lastVisible =
                layoutManager.findLastVisibleItemPosition()

            if (totalItems > 0 &&
                lastVisible >= totalItems - 5
            ) {
                viewModel.loadNextPage()
            }
        }
    }
)

The ViewModel's loading and end-of-pagination checks prevent most repeated requests from this listener. For complex interfaces, grid layouts, multiple data sources, and sophisticated retry behavior, prefer Paging 3's built-in demand-driven loading.

11. Introduction to Android Jetpack Paging 3

Android Jetpack Paging 3 is a library designed to load and display large datasets gradually from sources such as network APIs or local databases. It integrates with Kotlin coroutines, Flow, RecyclerView, Room, and Jetpack Compose.

Paging 3 provides built-in support for load states, retry, refresh, request deduplication, and in-memory caching. It also reduces the amount of custom code required to detect when more data is needed.

Official documentation: Android Paging Library Overview.

12. Main Components of Paging 3

13. Add the Paging Dependency

For a RecyclerView-based project using Kotlin coroutines, add the Paging runtime dependency to the app module. Use a stable version compatible with the project.

dependencies {
    val pagingVersion = "3.5.3"

    implementation(
        "androidx.paging:paging-runtime:$pagingVersion"
    )
}

The version shown is an example of a current Paging 3 release. Check the official release information and your project's compatibility requirements before choosing a version. Paging Compose projects also need the appropriate Paging Compose integration dependency.

14. Creating a PagingSource

A PagingSource defines how to load data for a particular key. In a page-number API, the key can be an integer page number, and the value is the data model returned for each record.

The example below assumes a repository with a function that retrieves one page from a remote API.

class ProductPagingSource(
    private val repository: ProductRepository
) : PagingSource<Int, Product>() {

    override fun getRefreshKey(
        state: PagingState<Int, Product>
    ): Int? {
        return state.anchorPosition?.let { position ->
            state.closestPageToPosition(position)?.let { page ->
                page.prevKey?.plus(1)
                    ?: page.nextKey?.minus(1)
            }
        }
    }

    override suspend fun load(
        params: LoadParams<Int>
    ): LoadResult<Int, Product> {
        val page = params.key ?: 1

        return try {
            val result = repository.getProducts(
                page = page,
                pageSize = params.loadSize
            )

            LoadResult.Page(
                data = result.items,
                prevKey = if (page == 1) null else page - 1,
                nextKey = if (result.hasMore) page + 1 else null
            )
        } catch (exception: IOException) {
            LoadResult.Error(exception)
        } catch (exception: HttpException) {
            LoadResult.Error(exception)
        }
    }
}

Import androidx.paging.PagingSource, androidx.paging.PagingState, java.io.IOException, and retrofit2.HttpException where applicable. The code assumes the repository throws these exceptions for network failures.

Important: Some APIs impose a maximum page size or use a different continuation mechanism. In that case, adapt the request parameters and next-key calculation to the backend's contract. The example's hasMore field is illustrative.

15. Configuring Pager and PagingConfig

The Pager combines the PagingSource factory with a PagingConfig. Its flow emits PagingData as pages are loaded.

class ProductPagingRepository(
    private val repository: ProductRepository
) {
    fun getPagedProducts(): Flow<PagingData<Product>> {
        return Pager(
            config = PagingConfig(
                pageSize = 20,
                initialLoadSize = 40,
                prefetchDistance = 5,
                enablePlaceholders = false
            ),
            pagingSourceFactory = {
                ProductPagingSource(repository)
            }
        ).flow
    }
}

Here, pageSize is the target number of items requested per page, initialLoadSize is the preferred initial load size, and prefetchDistance determines how close the user can get to loaded content before more data is requested. The source and API may influence the actual returned count.

16. Expose PagingData from a ViewModel

The ViewModel exposes a Flow of paginated products and caches the stream within its lifecycle. This helps preserve loaded data across configuration changes such as device rotation.

class ProductPagingViewModel(
    repository: ProductPagingRepository
) : ViewModel() {

    val products: Flow<PagingData<Product>> =
        repository.getPagedProducts()
            .cachedIn(viewModelScope)
}

Import androidx.lifecycle.ViewModel, androidx.lifecycle.viewModelScope, androidx.paging.cachedIn, and kotlinx.coroutines.flow.Flow. The repository instance should be supplied using your project's dependency-injection or ViewModel factory approach.

17. Displaying Paged Data with PagingDataAdapter

PagingDataAdapter is designed to display data from Paging 3 in RecyclerView. It requests more data as the user scrolls and uses a DiffUtil.ItemCallback to compare items.

class ProductPagingAdapter :
    PagingDataAdapter<Product, ProductViewHolder>(
        object : DiffUtil.ItemCallback<Product>() {
            override fun areItemsTheSame(
                oldItem: Product,
                newItem: Product
            ): Boolean {
                return oldItem.id == newItem.id
            }

            override fun areContentsTheSame(
                oldItem: Product,
                newItem: Product
            ): Boolean {
                return oldItem == newItem
            }
        }
    ) {

    override fun onCreateViewHolder(
        parent: ViewGroup,
        viewType: Int
    ): ProductViewHolder {
        val view = LayoutInflater.from(parent.context)
            .inflate(R.layout.item_product, parent, false)

        return ProductViewHolder(view)
    }

    override fun onBindViewHolder(
        holder: ProductViewHolder,
        position: Int
    ) {
        val product = getItem(position)

        if (product == null) {
            holder.bindPlaceholder()
        } else {
            holder.bind(product)
        }
    }
}

This example assumes that ProductViewHolder includes a bindPlaceholder() method for clearing or displaying a loading placeholder. Because PagingDataAdapter can return null for unloaded placeholder positions when placeholders are enabled, binding code should handle nullable items safely.

If placeholders are disabled, the adapter typically binds actual loaded items, but handling the nullable result remains a safe practice.

18. Collecting PagingData in an Activity or Fragment

Attach the adapter to RecyclerView and collect the ViewModel's Flow using the lifecycle-aware coroutine scope.

recyclerView.layoutManager =
    LinearLayoutManager(requireContext())

recyclerView.adapter = productPagingAdapter

viewLifecycleOwner.lifecycleScope.launch {
    viewLifecycleOwner.repeatOnLifecycle(
        Lifecycle.State.STARTED
    ) {
        viewModel.products.collectLatest { pagingData ->
            productPagingAdapter.submitData(pagingData)
        }
    }
}

This example is for a Fragment. Required imports include androidx.lifecycle.Lifecycle, androidx.lifecycle.lifecycleScope, androidx.lifecycle.repeatOnLifecycle, kotlinx.coroutines.flow.collectLatest, and the relevant Paging and RecyclerView classes.

Use collectLatest because a newer PagingData generation can replace an older one. The lifecycle-aware collection stops and restarts according to the Fragment's view lifecycle.

19. Handling Loading, Error, and Retry States

Paging 3 exposes load states through the adapter's loadStateFlow. The UI can use these states to display a progress indicator, an error message, a retry button, or an empty-state message.

productPagingAdapter.addLoadStateListener { states ->
    when (val refresh = states.refresh) {
        is LoadState.Loading -> {
            // Show initial loading indicator.
        }

        is LoadState.Error -> {
            // Show error message and retry option.
        }

        is LoadState.NotLoading -> {
            // Hide initial loading indicator.
        }
    }

    when (val append = states.append) {
        is LoadState.Loading -> {
            // Show a footer progress indicator.
        }

        is LoadState.Error -> {
            // Show footer error and retry option.
        }

        is LoadState.NotLoading -> {
            // Hide footer loading indicator.
        }
    }
}

Import androidx.paging.LoadState. The comments indicate where to update your own views. In a full implementation, use separate UI states for initial loading and loading additional pages so the existing list remains visible during append operations.

Retry a failed load with:

productPagingAdapter.retry()

Refresh the currently displayed data with:

productPagingAdapter.refresh()

retry() retries failed loads, while refresh() requests a new PagingData generation. If your app uses a network-plus-database architecture, refresh behavior should also be coordinated with the repository and RemoteMediator.

20. Integrating Paging with Room Database

Room can provide paginated data from a local database. Instead of returning every record at once, a DAO can return a PagingSource.

@Dao
interface ProductDao {

    @Query("SELECT * FROM products ORDER BY id ASC")
    fun getProducts(): PagingSource<Int, ProductEntity>
}

The example assumes that ProductEntity is a Room entity and that the table is named products. Room invalidates the relevant PagingSource when database changes affect the query, allowing the Paging library to reload the necessary data.

Create a Pager using the DAO:

val productFlow: Flow<PagingData<ProductEntity>> =
    Pager(
        config = PagingConfig(pageSize = 20),
        pagingSourceFactory = { productDao.getProducts() }
    ).flow

This approach is useful for large local datasets because the database can return records incrementally rather than requiring the application to load the entire table into memory.

21. Using RemoteMediator for Network and Database Data

RemoteMediator is useful when the application fetches records from a remote API but displays data from a local Room database. Room acts as the source of truth for the UI, while RemoteMediator fetches additional remote pages and saves them into the database.

Typical Data Flow

1. The UI collects paginated data from a Room-backed PagingSource.
2. When additional data is needed, Paging can invoke RemoteMediator.
3. RemoteMediator requests the next page from the remote service.
4. The retrieved records and any required paging keys are saved to Room.
5. Room invalidates the affected PagingSource.
6. The UI receives updated database-backed data.

This architecture can support offline access because the UI reads from the local database. The exact offline behavior depends on the data already cached and the app's refresh and synchronization strategy.

22. Strategies for Handling Very Large Datasets

• Load incrementally: Avoid downloading all records before rendering the first screen.
• Use server-side filtering: Let the API return only records matching the requested search or category when possible.
• Choose suitable page sizes: Balance request frequency against payload size and response time.
• Use stable ordering: Make sure the backend uses a consistent sort order for predictable pagination.
• Prefer cursor pagination for changing feeds when supported: This can reduce issues caused by records shifting between offset-based requests.
• Cache thoughtfully: Use Room or suitable in-memory caching to reduce repeated network requests.
• Keep item layouts efficient: Avoid deeply nested layouts and expensive work during binding.
• Load images asynchronously: Use a suitable image-loading library rather than downloading full images on the main thread.
• Cancel obsolete work: Stop requests that are no longer relevant, such as searches for an outdated query.
• Monitor memory and network use: Test with realistic dataset sizes and slower connections.

23. Search and Filtering with Pagination

Search and filtering must be coordinated with pagination. When the search query changes, previously loaded pages may no longer match the query. A new paging stream should be created for the new query instead of appending unrelated results to the existing list.

A typical ViewModel approach uses a query Flow and creates a Pager for the current query. The stream can use flatMapLatest so obsolete searches are cancelled when a newer query arrives.

val searchResults: Flow<PagingData<Product>> =
    searchQuery
        .debounce(300)
        .distinctUntilChanged()
        .flatMapLatest { query ->
            repository.searchProducts(query)
        }
        .cachedIn(viewModelScope)

This is a conceptual example. It assumes that searchQuery is a Flow of strings and that repository.searchProducts(query) returns a Flow of PagingData. Import the relevant Flow operators and Paging types, and define the query state and repository implementation in your project.

24. Common Pagination Errors and Solutions

25. Best Practices

• Prefer Android Jetpack Paging 3 for production applications with large or continuously updated datasets.
• Use manual pagination only when the requirements are simple and you can reliably manage loading, errors, and refresh behavior.
• Choose page-number, offset, or cursor pagination according to the backend API.
• Keep data access in a repository and expose UI-ready state through a ViewModel.
• Use PagingDataAdapter for RecyclerView-based paginated lists.
• Use cachedIn(viewModelScope) to share the paging stream across UI recreation.
• Implement meaningful loading and error states for both initial loading and additional-page loading.
• Do not assume that every API returns exactly the requested page size.
• Use Room and RemoteMediator when local caching and network synchronization are required.
• Test empty responses, final pages, failed requests, retries, rapid scrolling, and changing search terms.

26. Practical Project

Project: Build a Paginated Product Catalog.

1. Create a Product model with ID, name, price, and category.
2. Connect the app to a paginated product API or a simulated repository.
3. Display products using RecyclerView.
4. Implement a PagingSource and configure Pager with a suitable PagingConfig.
5. Expose the PagingData stream through a ViewModel.
6. Build a PagingDataAdapter with a DiffUtil.ItemCallback.
7. Add loading, error, and retry UI for page requests.
8. Add search and category filtering.
9. Optionally save products in Room and implement RemoteMediator for network-backed caching.
10. Test the app with a large dataset and simulate slow network conditions.

Expected result: The app loads the first group of products quickly, fetches more as the user scrolls, handles failures gracefully, and avoids keeping the entire remote dataset in memory.

27. Interview Questions and Answers

Q1. What is pagination?

Pagination divides a large dataset into smaller groups so the application can retrieve and display records gradually.

Q2. Why is pagination important in Android?

It helps reduce unnecessary memory use, network traffic, initial loading time, and UI work.

Q3. What is Android Paging 3?

It is an Android Jetpack library that supports incremental data loading, load states, retry, caching, and integration with Kotlin coroutines and Flow.

Q4. What is PagingSource?

PagingSource defines how to load a page of data from one source using a particular type of key.

Q5. What is Pager?

Pager combines a PagingConfig with a PagingSource factory and exposes a stream of PagingData.

Q6. What is PagingConfig?

PagingConfig defines paging behavior, including page size, initial load size, prefetch distance, placeholder support, and optional memory limits.

Q7. What is PagingDataAdapter?

It is a RecyclerView adapter designed to display PagingData and request additional data as needed.

Q8. What is RemoteMediator?

RemoteMediator coordinates remote loading with a local data source, commonly fetching API data and saving it into Room.

Q9. What is the difference between offset and cursor pagination?

Offset pagination skips a specified number of records, while cursor pagination continues from a token or key supplied by the backend.

Q10. How should an app handle a failed page request?

It should show an appropriate error state, allow retry, preserve already loaded data where possible, and avoid issuing uncontrolled duplicate requests.

28. Quick Revision

• Pagination: Loads data in smaller portions.
• Page-number pagination: Uses a page index and page size.
• Offset pagination: Uses a starting offset and a limit.
• Cursor pagination: Uses a continuation key.
• PagingSource: Loads data from one source.
• Pager: Creates the paginated data stream.
• PagingConfig: Controls page loading behavior.
• PagingDataAdapter: Displays paginated data in RecyclerView.
• RemoteMediator: Coordinates network loading with a local database.
• cachedIn(viewModelScope): Shares and caches paging data across configuration changes.

29. Official Learning Resources

• Android Developers: Paging Library Overview
• Android Developers: Load and Display Paged Data
• Android Developers: Paging from Network and Database

Conclusion

Pagination is an essential technique for building Android applications that handle large collections of data. Manual pagination can work for simple requirements, but Android Jetpack Paging 3 provides a more complete solution for incremental loading, retries, caching, and lifecycle-aware data streams. By combining PagingSource, Pager, PagingDataAdapter, and optionally Room with RemoteMediator, developers can build scalable and responsive interfaces without loading every record at once.

Learn more through the JustAcademy Android App Development course, register for a course demo, or visit the JustAcademy homepage.